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Abstract 

An artificial neural network vector quantizer is devel- 
oped for use in data compression applications such 
as Digital Video. Differential Vector Quantization is 
used to preserve edge features, and a new adaptive 
algorithm, known as Frequency-Sensitive Competitive 
Learning, is used to develop the vector quantizer code- 
book. To develop real time performance, a custom 
VLSI ASIC is being developed to realize the associa- 
tive memory functions needed in the vector quanti- 
zation algorithm. By using vector quantization, the 
need for Huffman coding can be eliminated, result- 
ing in superior performance against channel bit errors 
than methods that use variable length codes. 

1 Introduction 

Effective data compression algorithms are needed to 
reduce transmission bandwidth and storage space. In 
particular, there is a great deal of interest in the low 
bit rate coding of images. In this paper, we discuss 
the compression of digital video image data, which 
has become a central concern as HDTV standards be- 
gin to develop. One compression technique, Vector 
Quantization (VQ) [1, 2], has emerged as a power- 
ful technique that can provide large reductions in bit 
rate while preserving essential signal characteristics. 
In this paper we show that error-insensitive VQ en- 
coders can be constructed by employing entropy based 
VQ codebooks. 

The purpose of this paper is to describe the use 
and implementation of an Artificial Neural Network 
(ANN) Vector Quantizer. More specifically, we discuss 
the design of a real-time, edge-preserving Differential 
Vector Quantizer (DVQ) architecture. We discuss the 
use of an ANN algorithm to design VQ codebooks, and 
we anticipate that the use of the same ANN algorithm 
can be employed in adaptive DVQ coders. The partic- 
ular ANN algorithm we use is called Frequency Sen- 
sitive Competitive Learning (FSCL). This algorithm 


has been described in depth in previous publications 
[3,4], so only a brief presentation is given here. 

A locally-optimal vector quantization algorithm, 
proposed by Linde, Buzo, and Gray (LBG) [5], has 
been extensively employed in encoding both speech 
and images. However, studies have shown that, in 
many cases, the computational complexity of this al- 
gorithm restricts its use in real-time applications [1, 6]. 
The use of AN Ns to perform vector quantization has 
been proposed to overcome these limitations. 

The use of ANNs for vector quantization has a num- 
ber of significant advantages. First, ANNs are highty- 
parallel architectures and thus offer the potential for 
real-time VQ. Second, the large body of training tech- 
niques for ANNs can be adapted to yield new, and 
possibly better, algorithms for VQ codebook design. 
Third, in contrast to the batch training mode of al- 
gorithms based on the LBG algorithm [/], most ANN 
training algorithms are adaptive; thus, ANN based VQ 
design algorithms can be used to build adaptive vec- 
tor quantizers [8]. This is crucial in applications where 
the source statistics are changing over time. 

This paper is organized as follows. First, we briefly 
describe basic Vector Quantization techniques and dis- 
cuss ANN VQ techniques. We then describe the FSCL 
algorithm and show how the FSCL algorithm attempts 
to build a maximum-entropy codebook. Then, in Sec- 
tion 3, we describe how a VQ encoder can be viewed 
as an Associative Memory (AM) and discuss issues 
related to the design and implementation of an AM. 
This is followed by a short discussion on the Differen- 
tial Vector Quantization architecture which is used to 
minimize edge distortion. We then present, our exper- 
imental results in Section 5 where a FSCL codebook 
is used in a DVQ architecture to compress digital im- 
ages. 
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2 Vector Quantization and the FSCL 
Artificial Neural Network 

2.1 Basic Vector Quantization Concepts 

Vector quantization capitalizes on the underlying 
structure of the data being quantized. The space of 
the vectors to be quantized is divided into a number of 
regions of arbitrary volume and a reproduction vector 
is calculated for each region. Given any data vector 
to be quantized, the region in which it lies is deter- 
mined and the data vector is then represented by the 
reproduction vector for that region. Instead of trans- 
mitting or storing a given data vector, a symbol which 
indicates the appropriate reproduction vector is used. 
I his can result in considerable savings in transmission 
bandwidth, albeit at the expense of some distortion. 

More formally, vector quantization maps arbitrary 
data vectors to a binary representation or symbol. 
Thus, the VQ mapping is from a ^-dimensional vector 
space to a finite set of symbols, M. Associated with 
each symbol m £ {M} is a reproduction vector x m . 
The encoding of the data vector x to the symbol in is 
a mapping, 

V Q : x = (x*i , x 2l • • ■ , Zfc) — ► m 

where m £ {M} and the set M has size M . Assuming 
a noiseless transmission or storage channel, m is de- 
coded as x m , the reproduction vector associated with 
the symbol m. The collection of all possible repro- 
duction vectors is called the reproduction alphabet or 
more commonly the codebook. Since there are M el- 
ements in the set M, there are M possible entries in 
the codebook. Once the codebook is constructed and, 
if necessary, transmitted to the receiver, the encoded 
symbol m acts as an index into the codebook. Thus, 
the rate , R , of the quantizer is R — log 2 M bits per in- 
put vector. Since each input vector has k components, 
the number of bits required to encode each input vec- 
tor component is R/k. 

Since each data vector must be ultimately repre- 
sented as one of the codebook entries, the composition 
of the codebook determines the overall performance of 
the system. A number of different performance crite- 
ria can be used to determine an optimal codebook, 
for example, in image transmission applications the 
usual objective is to minimize the overall distortion in 
the signal due to VQ. Thus the design criterion used to 
design an optimal codebook is the minimization of the 
average distortion in encoding vectors using the code- 
book. Another possible criterion is to maximize the 
entropy of the codebook, i.e., to ensure that each of 
the codewords is used equally frequently in encoding 


the data. This is a very useful criterion in develop- 
ing ANN training algorithms for VQ design because 
maximum entropy codebooks can be employed with- 
out the use of Huffman codes, thus reducing encoder 
sensitivity to channel errors. Finally, an alternative 
criterion is to use a distortion measure that incorpo- 
rates expected responses of the human visual system 
to differences in intensity values and motion. 

Given a performance criterion, the VQ codebook 
design process involves the determination of a code- 
book that is optimal with respect to this criterion. 
This normally requires knowledge of the probability 
distribution of the input data. Typically, however, 
this distribution is not known, and the codebook is 
constructed through a process called training. During 
training, a set of data vectors that is representative of 
the data that will be encountered in practice is used 
to determine an optimal codebook. 

During the training process, a distortion measure, 
d(x,x) is typically used to determine which data 
points are to be considered as being in the same re- 
gion. The distortion measure can be viewed as the cost 
of representing x as x. By determining which training 
data vectors lie in the same region, the dimensional 
data space is partitioned into cells. All of the input 
vectors that fall into a particular cell are mapped to a 
single, common reproduction vector. 

2.2 Motivations for the use of ANN VQs 

Unfortunately, the VQ training and encoding pro- 
cesses are computationally expensive. Moreover, most 
of the algorithms currently used for VQ design are 
batch mode algorithms [5], and need access the entire 
training data set during the training process. Using 
ANN adaptive techniques, it is possible to realize an 
adaptive VQ coder in which codewords are modified 
based on the arrival of each new training vector. 

2.3 The FSCL Algorithm 

The Frequency - Sensitive Competitive Learning 
(FSCL) algorithm is an unsupervised ANN consist- 
ing of two layers. The input layer nodes transmit the 
input vector elements to each of the nodes in the out- 
put layer. In the output layer, known as the winner - 
take-all layer , each node receives inputs from all of 
the input nodes. The weighted interconnections be- 
tween these two layers are considered the exemplar , or 
weight vectors and are used for selecting the winner 
node. The winning node is selected on the basis of 
a modified distortion measure for each of the output 
layer nodes. The FSCL codebook design algorithm 
used in a training phase is discussed below. 
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One of the motivations for the Frequency-Sensitive 
Competitive Learning (FSCL) network is to overcome 
the limitations of simple competitive learning network 
while retaining its computational advantages. One of 
the main problems with CL networks is that some of 
the neural units may be under-utilized, the learning 
algorithm for the FSCL network keeps a count of how 
frequently each neural unit is the winner. This in- 
formation is used to ensure that, during the training 
process, all neural units are modified an approximately 
equal number of times. This yields a codebook that, 
on average, utilizes all of codewords equally. Conse- 
quently, the use of variable-length Huffman codes is 
unnecessary because no additional compression will be 
achieved through their use. 

To solve the under-utilization problem and obtain 
an equiprobable codebook, the FSCL Algorithm uses 
a fairness function, T{ui ), which is a function of the 
local update counter , iz,, and is chosen to ensure the 
utilization of all the nodes in the winner-take-all layer. 
The motivation and use of fairness function has been 
discussed in previous papers [3, 4]. 

Finally, if the codewords are indexed or labeled such 
that codewords which are close in Hamming distance 
are also close in the chosen distortion criteria (e.g., 
absolute distance) then the resulting encoding archi- 
tecture will be relatively insensitive to transmission er- 
rors. This is because random bit errors in the channel 
will result in reproduction vectors which are “close” 
in the distortion criteria chosen by the codebook de- 
signer. 

2,3.1 The FSCL Training Phase 

In the training phase , the exemplar vectors in the 
winner- take-all layer are adjusted adaptively to statis- 
tically reflect the distribution of the training vectors. 
A training vector is applied to the input layer and 
compared to all of the exemplar vectors in the winner- 
take-all layer. Upon the completion of the compar- 
isons, one node in the winner-take-all layer is selected 
to be the winner and the rest of the nodes are inhib- 
ited. The selection of the winner node depends on the 
product of the fairness function, ^(u,), and the dis- 
tortion measure. The distortion measure of each input 
training vector is calculated with respect to the exem- 
plar vector. Common approaches for measuring the 
distortion of the input and exemplar vectors are dot 
product, Euclidean distance, and absolute distance. 

The exemplar vector of the winning node is ad- 
justed, by an amount specified by the learning rate, 
so as to more closely represent the input vector. Fi- 
nally, the winner node increments its update-counter, 


Ui, and then the next training vector is presented to 
the network. Each node in the winner-take-all layer 
has a private update-counter, and the counters are 
used to influence the selection of the winner nodes. 
As a result, infrequently used exemplar vectors are 
adjusted, even if they had a larger distortion measure 
than other exemplar vectors [9]. 

When the learning phase is completed, all of the 
weight vectors in the winner-take-all layer are adapted, 
and the FSCL-derived codebook can be used in the 
encoder. As previously noted, the use of the modified 
distortion measure insures that the codewords are up- 
dated approximately the same number of times, thus 
maximizing the entropy of the codebook. 

2.3.2 The Encoding and Decoding Phase 

After the codebook has been constructed, input data 
vectors are coded by comparing each data vector with 
each of the codewords and then transmitting (or stor- 
ing) the index of the (winning) codeword which yields 
the minimum distortion. Finding the minimum dis- 
tortion codeword is a time-consuming task, but this 
operation is inherently parallel. The parallel hard- 
ware we have developed for the encoding is discussed 
in Section 3, and uses winner-take-all circuits in an 
Associative Memory encoder. Note that the winner- 
take-all circuits are used during training and encoding. 
However, only during training does adaptation of the 
codewords occur. 

To decode the datum the receiver uses the received 
index to access a copy of the codebook to determine 
the reproduction vector used to represent the original 
data. This is a simple look-up process, and can be 
done without special hardware support in conventional 

RAM. 

3 Vector Quantization as Associative 
Memory 

VQ can be thought of as the process taking an input 
vector and matching it to the closest vector out of 
a set of vectors. Once the closest match is found, the 
index of the match is transmitted to the receiver. This 
process can also be viewed as an Associative Memory 
(AM), where the index is associated with a particular 
codeword. 

As an example of an associative memory which uses 
Hamming distance and has very fast matching capa- 
bilities, consider the following design. The AM cell, 
shown in Figure 1, is a simple variant of the standard 
static RAM cell. 
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WORD 
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Figure 1: An Associative Memory cell. The transistors 
M7, M8 and M9 are the only additions to a standard 
static RAM. Their function is to draw the MATCH 
line low when the value on the bit and BIT lines match 
the value stored in the static RAM core. 


When in a matching mode, i.e. any time the word 
line is not asserted, if the value on the BIT line matches 
that stored by transistors Ml and M3, the transis- 
tor M7 becomes active and M9 is turned on and the 
match line is drawn down. Similarly, if the BIT line 
matches the value stored in transistors M2 and M4, 
the transistor M8 becomes active and again M9 turns 
on. 

The AM cells are arranged in an array such that the 
words to be matched all share the same MATCH line, 
as shown in Figure 2. 
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Figure 2: An array of associative memory cells, with 
their corresponding closest match circuitry. The cir- 
cuitry which determines the closest match is a winner- 
take-all network which grows linearly with the number 
of words to match. 


via the BIT and bit lines, the AM cells begin to sink 
current if the value stored in the AM matches that 
presented on the BIT and BIT lines. The circuitry at 
the bottom then determines which current coming in 
is the highest. The only Iout with any output current 
is the Iout f° r which the corresponding MATCH line 
that is the largest. This Iout is then used to gate the 
value of the given AM column to the output buffers. 

4 Differential Vector Quantization 

Differential Pulse Code Modulation (DPCM) can be 
used to perform quantization of image data in a 
CODEC (COder-DECoder) architecture. One exam- 
ple can be found in [10] which is based on DPCM 
but also utilizes a nonuniform quantizer and multilevel 
Huffman coding to reduce the data rate substantially 
below that achievable with straight DPCM. As a re- 
sult of variable length coding, the compression ratio is 
different for different images depending on the statis- 
tics of that particular image. Furthermore, because of 
the different codeword lengths it is very difficult to de- 
tect and compensate for transmission errors even if an 
end-of-line reset is used to re-sychronize the encoder 
and the decoder. 



Figure 3: Block Diagram for Differential Vector Quan- 
tizer 


At the top of each column of AM cells is a current 
supply, which drives the match line. In match mode 
as the value to be matched is shown to the column 
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Differential Vector Quantization (DVQ) (see Fig- 
ure 3) incorporates the desirable qualities of both 
Differential Pulse Code Modulation (DPCM) and 









VQ [11]. In DVQ, instead of scalar quantizing the 
scalar difference values (as in DPCM), vectors of differ- 
ence values (difference tiles) are vector quantized and 
their codeword indices transmitted. At the receiver, 
the indices are used to look up the reconstruction dif- 
ference vector which is then added to the predicted 
difference vector. 

Because the VQ coder uses the FSCL derived code- 
books, over a large sample of images the codewords are 
utilized relatively equally. Consequently, fixed length 
codes are used and the codec does not need to use 
synchronizing codes. Furthermore, as discussed ear- 
lier, by arranging the codewords so that codewords 
which are Hamming-close are also distortion-close we 
can achieve a significant level of error insensitivity, as 
the results in Section 5 clearly show. 

5 Results 

Table 1 shows the MSE obtained when DVQ codec 
using a FSCL codebook was used to compress eight 
images. For comparison, the MSE for a version of a 
DPCM codec, as described in [10], is also given. As 
can be seen, the DPCM codec yields MSEs which are 
lower than those of the DVQ codec. However, as can 
be seen in Figures 4 and 5 the images are virtually 
indistiguishable (these images are indistinguishable at 
full resolution, as well). Furthermore, as shown in Fig- 
ure 6, the DVQ codec is significantly more robust to 
channel errors. In the left picture of Figure 6 line 
resynchronization was used in an attempt to mini- 
mize errors to one line of the image. Note that, even 
with resynchronization, the method results in trun- 
cated lines. Furthermore, when errors occur in the 
synchronizing codeword, lines are missed entirely. 

6 Conclusions 

We have presented an ANN based DVQ codec which 
is well suited for use in data compression applications 
such as Digital Video. There are three novel aspects 
of this work. First, Differential Vector Quantization is 
used to preserve edge features. Second, a new adaptive 
algorithm, known as Frequency-Sensitive Competitive 
Learning, is used to develop a vector quantizer code- 
book that eliminates the need for Huffman coding. 
Finally, in order to realize real time performance, a 
custom VLSI ASIC is being constructed to perform 
the associative memory functions needed in the vec- 
tor quantization algorithm. The resulting codec ex- 
hibits greater compression and superior performance 
against channel bit errors than methods that use vari- 
able length codes. 
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Table 1: MSE of NASA Codec and FSCL Differential Vector Quantization Algorithms 



Bits Per Pixel (BPP) 

p MSE: No Errors \ 

MSE: BER = 

1/1000 

Picture 

codec 

dvq 128 

dvq 256 

codec 

dvq 128 

dvq 256 

codec 

dvq 128 

dvq 256 

bird 

2.56 

1.75 

2.00 

6.5 

20.2 

14.5 

2152 

70.2 

71.6 

everest 

2.17 

1.75 

2.00 

5.0 

16.8 

12.6 

981 

44.1 

34.3 

fruityf 

2.30 

1.75 

2.00 

5.9 

16.5 

11.9 

604 

67.1 

71.0 

hail 

2.97 

1.75 

2.00 

7.7 

17.8 

12.7 

1010 

71.1 

63.0 

kittyj 

2.75 

1.75 

2.00 

5.9 

14.0 

10.1 

964 

65.7 

47.9 

lenna 

2.37 

1.75 

2.00 

5.3 

11.4 

8.3 

1213 

47.7 

34.6 

mandril 

4.05 

1.75 

2.00 

13.1 

89.0 

66.4 

3107 

305.7 

201.1 

planet 

2.09 

1.75 

2.00 

4.6 

13.9 

10.0 

1113 

56.6 

61.6 

scenef 

2.87 

1.75 

2.00 

7.9 

22.6 

16.3 

1993 

75.7 

75.7 

»ft 

2.89 

1.75 

2.00 

9.5 

57.1 

41.9 

3008 

159.9 

139.7 


fThese images were used in training the vector quantizer. 
Error MSE’s averaged over 3 trials. 




Figure 4: Original image (left) and reconstructed image using the NASA codec algorithm (right) 


ORIGINAL PAGE 

BLACK AND WHITE PHOTOGRAPH 


ORIGINAL PAGE IS 
OF POOR QUALITY 


100 





Figure 5: Reconstructed image using differential vector quantization, FSCL with 128 codewords (left) and 256 
codewords (right) 



Figure 6: Reconstructed image with a noisy transmission channel (1 error per 1000 bits), NASA codec algorithm 
(left) and differential vector quantization using FSCL with 256 codewords (right) 
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